In massive Multiple-Input-Multiple-Output (MIMO) system, as the increse of antenna number, traditional detection algorithms have lower performance, higher complexity, and they are not suitable for high order modulation. To solve the problem, based on the idea of shadow domain, a search tree detection algorithm combining Quadratic Programming (QP) and Branch and Bound (BB) algorithm was proposed. Firstly, with QP model constructed, the unreliable symbols from solution vector of first-order QP algorithm were extracted; then, BB search tree algorithm was applied to the unreliable symbols for the optimal solution; meanwhile three pruning strategies were proposed to reach a compromise between complexity and performance. The simulation results show that the proposed algorithm increases 20 dB performance gain compared with the traditional QP algorithm in 64 Quadrature Amplitude Modulation (QAM) and increases 21 dB performance gain compared with QP algorithm in 256 QAM. Meanwhile, applying the same pruning strategies, the complexity of the proposed algorithm is reduced by about 50 percentage points compared with the traditional search tree algorithm.